Thanks, but llama.cpp maintainers made it very clear that I should add CPU implementation first and implementation for other backends (CUDA etc) in separate following PRs, so I'm not going to include a CUDA implementation (mine, yours or any) in my current CPU lightning indexer PR branch. Sorry, but you can try later after it's merged. Wish your luck!

This is such an amazing work

The master fork of llama.cpp

1. is much slower for prompt processing
2. can't handle more than 64k context on a 5090
3. has amnesia (Eval bug: Deepseek V4 Flash forgets everything ggml-org/llama.cpp#25171 & Eval bug: DeepSeek V4 Flash forgot context when hit cache or use checkpoint in long context ggml-org/llama.cpp#25259)

Your work fixes all of them

Thanks, but llama.cpp maintainers made it very clear that I should add CPU implementation first and implementation for other backends (CUDA etc) in separate following PRs, so I'm not going to include a CUDA implementation (mine, yours or any) in my current CPU lightning indexer PR branch. Sorry, but you can try later after it's merged. Wish your luck!

Got it - wishing you luck on them accepting your cpu indexer pr soon then!

Report: works multi-GPU (2×3090, 1M ctx) — but llama-server defaults crash-loop on ANY hardware (-np 1 fixes it)

Built deepseek-lid-cuda @ 6164d85 on Linux (Ubuntu 24.04, CUDA 13.3 toolkit /
13.0 driver, arch 86). 2× RTX 3090 layer-split + CPU expert offload, 64GB DDR5,
antirez IQ2XXS chat-v2 quant, F16 KV.

Bench (all clean exits, llama-batched-bench -npl 1):

Flat decode 256K→1M reproduces. No kernel issues across the layer-split boundary.

The finding: llama-server with default --parallel -1 (auto slots) crash-loops
at startup — graph reserve goes down the unfused indexer path:

ggml_backend_cuda_buffer_type_alloc_buffer: allocating 398314.19 MiB on device 0: cudaMalloc failed: out of memory
ggml_gallocr_reserve_n_impl: failed to allocate CUDA0 buffer of size 417662695680
graph_reserve: failed to allocate compute buffers

(256K ctx, ub2048.) Not multi-GPU-related — reproduced with CUDA_VISIBLE_DEVICES=0
(single card, 64K ctx): same fallback, ~231 GiB reserve attempt. So default
llama-server fails on any hardware; every validation so far (mine included) used
llama-cli/llama-bench, which are single-sequence and never hit it.

Workaround: -np 1 → server reserves the fused path (~3.2 GiB) and works —
running behind Open WebUI at 13.8 t/s now. Looks like the fused wiring in
deepseek4.cpp doesn't cover multi-sequence ubatches, so multi-slot worst-case
reserve falls back to the unfused chain. Happy to test a fix or provide more logs.

Two multi-GPU notes for the docs: the -ot preset needs GPU expert blocks straddling
both cards' contiguous layer ranges (lowest-N-blocks piles everything onto device 0
→ OOM at load), and each card allocates the full compute buffer from your formula,
not half.

@jonearth Indeed I see weird things happening with multiple parallel sequences (while -np 1 runs fine). Thanks for bringing this to my attention.

@spencer-zaid I've updated my CPU lightning indexer implementation in ggml-org#24231, hopefully it will be merged soon. Note that there were some API changes - scale factors parameters were removed and f16 mask parameter was added. I added some test-backend-ops tests in the CPU implementation so you can test your CUDA implementation with this and measure the performance.

My CUDA implementation will follow: ggml-org#25545 but maybe yours if faster, I guess it's worth checking.